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Abstract 

Background: In 2005, the International Patient Decision Aids Standards Collaboration identified twelve quality 
dimensions to guide assessment of patient decision aids. One dimension — the delivery of patient decision aids on 
the Internet — is relevant when the Internet is used to provide some or all components of a patient decision aid. 
Building on the original background chapter, this paper provides an updated definition for this dimension, outlines 
a theoretical rationale, describes current evidence, and discusses emerging research areas. 

Methods: An international, multidisciplinary panel of authors examined the relevant theoretical literature and 
empirical evidence through 2012. 

Results: The updated definition distinguishes Internet-delivery of patient decision aids from online health 
information and clinical practice guidelines. Theories in cognitive psychology, decision psychology, communication, 
and education support the value of Internet features for providing interactive information and deliberative support. 
Dissemination and implementation theories support Internet-delivery for providing the right information (rapidly 
updated), to the right person (tailored), at the right time (the appropriate point in the decision making process). 
Additional efforts are needed to integrate the theoretical rationale and empirical evidence from health technology 
perspectives, such as consumer health informatics, user experience design, and human-computer interaction. 
Despite Internet usage ranging from 74% to 85% in developed countries and 80% of users searching for health 
information, it is unknown how many individuals specifically seek patient decision aids on the Internet. Among the 
86 randomized controlled trials in the 2011 Cochrane Collaboration's review of patient decision aids, only four 
studies focused on Internet-delivery. Given the limited number of published studies, this paper particularly focused 
on identifying gaps in the empirical evidence base and identifying emerging areas of research. 

Conclusions: As of 2012, the updated theoretical rationale and emerging evidence suggest potential benefits to 
delivering patient decision aids on the Internet. However, additional research is needed to identify best practices 
and quality metrics for Internet-based development, evaluation, and dissemination, particularly in the areas of 
interactivity, multimedia components, socially-generated information, and implementation strategies. 
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Background 

In 2005, the International Patient Decision Aids Standards 
(IPDAS) Collaboration identified twelve dimensions that 
could guide the assessment of the quality of patient deci- 
sion aids (PtDAs) [1,2]. One of these dimensions — the 
delivery of PtDAs on the Internet— is relevant when the 
Internet is used to provide some or all components of a 
PtDA. This dimension raises a unique challenge in asses- 
sing quality, requiring an integration of relevant definitions, 
theories, and evidence from decision science, medicine, 
medical/consumer health informatics, user experience 
design, human-computer interaction, health services 
research, and implementation science. Therefore, the aims 
of this paper are to provide an updated definition for this 
dimension, outline the current theoretical rationale for 
Internet-delivery as an important aspect of quality assess- 
ment, identify the gaps in empirical evidence, and discuss 
emerging research areas needed for assessing quality. 

An updated definition 

"Delivering patient decision aids on the Internet" is 
defined as the process of using the Internet to provide 
some or all components of a PtDA to help individuals (e. 
g., patients, caregivers, proxy decision makers, etc.) 
involved in the process of choosing between two or more 
medically-appropriate healthcare options (e.g., preference- 
sensitive care). This definition is intended to differentiate 
Internet-delivered PtDAs that support shared decision 
making in preference-sensitive care from online health 
education websites, clinical practice guidelines, clinicians' 
decision support, or expert systems [3]. 

This broad conceptual definition covers a range of 
operational approaches — from providing a downloadable 
copy of a paper-based PtDA, to streaming a video-based 
PtDA online, to providing interactive decision support 
websites that tailor information and support to the needs 
and preferences of each individual decision maker. It is 
important to note that delivery of a PtDA on the Internet 
may be only one component of a larger decision support 
program (e.g., viewing a PtDA website coupled with a 
phone call from a decision coach). Internet-delivered 
PtDAs may also be used during, in conjunction with, or 
separate from a clinical consultation. However, the pur- 
pose of this definition is to provide a perspective from 
which patients, families, clinicians, decision aid developers, 
and policy makers can evaluate the quality of the manner 
in which a PtDAs is delivered on the Internet. 

Updated theoretical justification 

The current rationale for delivering PtDAs on the Internet 
draws on a range of disciplines, including psychology, edu- 
cation, and implementation science. Several theories in cog- 
nitive psychology, decision psychology, and communication 
emphasize the value of using the Internet to provide broad 



and long-term dissemination of information that can be 
targeted and tailored to patients' needs and preferences. 
For example, the Health Belief Model emphasizes the 
importance of providing tailored information to motivate 
active engagement in health care [4]. Similarly, interactive 
deliberative tasks foster self-efficacy and lead to increased 
engagement (Social Cognitive Theory) [5]. The Elaboration 
Likelihood Model proposes that people attend to and 
actively process information more if it is perceived as per- 
sonally-relevant [6]. The Theory of Goal Setting and Perfor- 
mance supports the role of interactivity in producing 
tailored actionable personal health goals [7]. Finally, the 
Stages of Change Theory supports the value of having up- 
to-date information and support readily accessible over 
time, so that patients can revisit the information and sup- 
port across the progression of disease and multiple iterative 
decision-making cycles [8]. 

From the perspective of the individual patient, theories 
of active learning, discovery learning, and social learning 
inform the design of Internet-delivered decision support 
tools that effectively inform and educate [9]. Behavior- 
ism emphasizes the need for measurable behaviors to 
confirm learning [10]. Information quizzes can reinforce 
awareness and facilitate realistic expectations of out- 
comes. Cognitive psychology emphasizes the internal 
processes of memory, motivation, thinking, and reflec- 
tion as essential in understanding how new information 
fits within one's existing knowledge and experience [11]. 
Interactive activities such as clarifying one's personal 
attitudes about the individual risks and benefits of each 
treatment option can reinforce comprehension and per- 
sonalization. Furthermore, constructivism builds on 
these theories to incorporate observation, processing, 
and interpretation of information in building one's per- 
sonal reality [12]. Ally integrates these three theoretical 
constructs to describe how health information provided 
online should be designed to help individuals learn 
about the "what" (behaviorist), "how" (cognitive psychol- 
ogist), and "why" (constructivist) of their options [9,13]. 

At the population level, dissemination and implemen- 
tation theories emphasize the importance of establishing 
effectiveness for the maximum number of people. The- 
ories of innovation, knowledge translation, and organiza- 
tional change may inform strategies for effective use of 
web-based PtDAs within medical centers and associated 
clinics [14,15]. Health equity frameworks support the 
development of technology-supported interventions to 
extend the reach of medical centers into rural or under- 
served communities. Community organizing models 
such as Reach Effectiveness Adoption Implementation 
(RE-AIM), Precede-Proceed, etc., can guide consumer- 
informed design approaches, particularly for chronic dis- 
ease self-management and for implementation within 
community programs [16,17]. 
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Updated empirical evidence 

The theoretical justification outlined above supports the 
idea of using the Internet to deliver PtDAs, although there 
are notable gaps in the empirical evidence. Three primary 
barriers must be addressed before a comprehensive sys- 
tematic review of the effectiveness and quality of Internet- 
delivery of PtDAs can be undertaken. 

First, greater definitional and methodological clarity is 
needed. Definitions are needed to distinguish between 
PtDAs made available for download from the Internet, 
those adapted for use on the Internet, and those designed 
and tested as used on the Internet. Efforts are also 
needed to distinguish among several overlapping terms 
across disciplines. For example, Internet-delivered PtDAs 
may be tailored, targeted, customized or personalized 
(see Additional file 1 Table SI) at several different levels, 
including: a) at the technology level, if they provide infor- 
mation and technology features based on users' needs 
and preferences (consumer health informatics, user 
experience design) [18]; b) at the decision support level, 
based on patients' decision-making needs and preferred 
deliberative styles (decision science) [3,19]; and c) at the 
dissemination level, based on cultural, age-related, or 
decision-making roles (e.g., patient, caregiver, legal 
proxy, etc.) and membership in user groups (implemen- 
tation science) [20]. Furthermore, quality metrics may 
differ by disciplinary perspective; different disciplines 
focus on assessing the usability of the Internet-delivered 
tool for supporting patients' decision making [21], on the 
usability of features of the user interface [22], and/or on 
the usability of using Internet delivery to extend patients' 
decision support programs into the community [17,22]. 

Second, empirical studies are needed that specifically 
assess the need for, and the effectiveness of, Internet-deliv- 
ery of PtDAs. Since the IPDAS quality criteria were first 
published [1,2], Internet usage studies have reported con- 
tinued increase in health information-seeking on the Inter- 
net [22-28], but have not specifically assessed the usage 
rates for PtDAs. Online libraries of PtDAs provide access 
to an increasing number of PtDAs [29], but few PtDAs 
have been developed using health technology research 
methods (such as user experience design, human compu- 
ter interaction, etc.) or have been evaluated for effective- 
ness as used on the Internet. A Cochrane Collaboration 
review of randomized controlled trials of the effectiveness 
of PtDAs began in 1996, and was recendy updated in 2011 
to include trials published through December 2009 [30]. 
However, Internet-delivery of PtDAs was not the focus of 
that review. Accordingly, a comprehensive systematic 
review of decision science, health technology, and imple- 
mentation science literature specifically focusing on the 
Internet-delivery of PtDAs is needed. 

Lastly, since Internet-delivery involves both human 
and computer factors, further empirical evidence is 



needed from multiple disciplinary perspectives. From 
the human factors perspective, the recent growth in 
making PtDAs available on the Internet has provided 
some evidence from the fields of decision psychology, 
health education, and medicine. Studies are emerging 
regarding the development and/or field testing of new 
Internet-delivered PtDAs, but, at this point, few rando- 
mized controlled trials of effectiveness have been com- 
pleted. From the Internet-delivery perspective, evidence 
from information technology perspectives— such as 
medical/consumer health informatics, user experience 
design, and human-computer interaction — need to be 
applied and tested in the context of patient decision 
support. For example, some online PtDAs may produce 
personalized risk estimates based on patients' clinical 
profiles (as in the field of medical informatics). Others 
may gather patients' reported informed, values-based 
preferences into health information systems (as in the 
field of consumer health informatics). Furthermore, 
attention must be paid to human-computer interaction 
and evidence about user experiences with design must 
be gathered, in order not only to identify "best prac- 
tices" in the design of Internet-based PtDAs but also to 
inform quality assessment measures and guidelines. At 
present, any attempt to carry out a full systematic 
review of Internet-based PtDAs is limited by these three 
barriers; accordingly, the following sections at least 
attempt to outline what is currently known and to high- 
light the gaps in empirical evidence regarding Internet- 
delivery of PtDAs. 

Use of the Internet for decision support 

For 2007-2011, http://Worldbank.org indicates that an 
average of 74.2% of individuals in the United States, 76% in 
Australia, 81% in Canada, 83% in Germany, and 85% in the 
United Kingdom use the Internet [23]. In 2011, the world- 
wide population of Internet users exceeded 3 billion (35% 
of the 7 billion world population), with over 1.8 billion 
homes receiving direct Internet access. In developing coun- 
tries, 25% of individuals have a computer, and 20% have 
Internet access. A 2011 survey in the United States [24] 
indicated that 80% of Internet users search for health infor- 
mation, making it the third most popular online activity 
(behind using e-mail and using search engines). Most of 
these individuals search for information about a specific 
disease (66%) or treatment procedure (56%). 

Similarly, the 2005-2010 International Telecommunica- 
tion Union estimates mobile phone usage rates at 90% of 
individuals in Korea, 83% in AustraUa, 75% in China, 59% 
in Italy, 56% in the United Kingdom, 38% in the Nether- 
lands, and 30% in Canada [26]. By 2011, the number of 
mobile-broadband subscriptions increased to 1.2 billion 
worldwide, with 90% of those subscriptions covered by 2G 
mobile-cellular networks and 45% covered by 3G networks. 



Hoffman ef al. BMC Medical Informatics and Decision Making 2013, 13(Suppl 2):S13 
httpy/www.biomedcentral.coni/1 472-6947/1 3/S2/S1 3 



Page 4 of 12 



Studies by the Pew Internet & American Life Project [27] 
reported a continuation in the upward trend in mobile 
phone usage in the U.S. Over 83% of U.S. adults reported 
owning a mobile phone, with 35% using a "smart phone" 
to access the Internet. Mobile Internet usage was most 
common among individuals who were 18-29 years old, 
African American, and Latino. Qualitative analyses sug- 
gested a growing preference for social connectivity (e.g., 
high rates of texting and social websites), and for searching 
for health information on the Internet prior to making an 
appointment with a doctor. 

In 2007, the DECISIONS telephone survey asked 
patients about their memory of using the Internet for nine 
selected medical decisions within the past two years [28]. 
The survey sample consisted of English-speaking U.S. 
adults (N = 2575) who, within the last two years, had 
talked with a clinician about, or had undergone, a proce- 
dure for one of nine decisions about medications (high 
blood pressure, high cholesterol, depression), screening 
(colorectal, breast, prostate), or surgery (cataracts, lower 
back pain, hip/knee replacement). An average of 28% of 
participants self-reported using the Internet related to 
these nine health decisions within the previous two years. 
Percentages varied from 17% for breast cancer to 48% for 
hip/knee replacement. Internet use decreased with age 
and increased by income level. Participants rated the Inter- 
net as their second-most important information source 
(their clinician being the first). Taken together, these stu- 
dies confirm the continued rise in rates of Internet use, 
health information-seeking, and Internet searching related 
to medical decisions. However, due to the lack of clarity in 
terminology and the limited assessment efforts, statistics 
regarding how many individuals seek PtDAs on the Inter- 
net remain unknovm. 

Inventories of PtDAs delivered on the Internet 

Multiple inventories of PtDAs are available on the Internet, 
but "Internet-delivery" is currently operationalized in sev- 
eral ways. For example, the Ottawa Hospital Research 



Institute maintains a Patient Decision Aid Library Inven- 
tory that provides browseable lists of PtDAs by clinical 
topic, as well as contact information and IPDAS quality 
scores [29]. As of October 2012, this inventory contained 
270 PtDAs, 198 of which were available in some form on 
the Internet. Currently, most of these PtDAs are available 
as brochures, worksheets, or videos that patients can 
download and use offline. Some have been adapted for 
viewing online. More recently, a few have been specifically 
designed for interactive use on the Internet. These differ- 
ences may represent sub-categories of Internet delivery, for 
which different evaluative measures are appropriate to 
assess their quality as used on the Internet (see Tables 1, 2 
and Additional file 1 Table SI). 

Studies of Internet delivery of patient decision aids 

While a systematic review of Internet delivery of PtDAs 
remains to be done, the 2011 update of Cochrane Colla- 
boration's review of the effectiveness of PtDAs includes 
four initial randomized controlled trials that used the 
Internet. [30] This systematic review included randomized 
controlled trials that a) were published up to December 
2009, b) compared a PtDA with usual care or an alterna- 
tive intervention, and c) involved individuals who were 
actively making a treatment or screening decision. 
Among the 86 studies that met these criteria, four studies 
tested Internet delivery approaches. Table 3 provides a 
summary of each study's clinical topic, study design, and 
primary results. 

Results of these four randomized controlled trials indi- 
cate that using an Internet-delivered PtDA: a) improved 
knowledge scores; b) was rated similarly on effort needed, 
convenience, and satisfaction; and c) had variable effects 
on preferences for screening. While these studies are the 
first randomized controlled trials to include Internet- 
delivery, initial results must be interpreted cautiously. 
Potential confounding variables were not addressed 
explicitly in the study reports, such as whether patients 
were involved in initial decision support and technology 



Table 1 Development Characteristics of PtDAs Delivered on the Internet [27-30,72,73] 



Internet-delivered 
PtDAs 

Internet-available 
PtDAs 

Internet-adapted 
PtDAs 



Internet-based 
PtDAs 



Includes all PtDAs for which some or all parts are delivered using the Internet 

May include PtDAs that were initially developed and tested in other formats (e.g., paper, audio, or video), then made 
available on the Internet. For example, several PtDA brochures were originally created as paper worksheets, and then made 
available online for individuals to download, print, and complete. 

May include PtDAs created in other formats that were purposefully adapted to allow individuals to use them directly on the 
Internet. Examples include adapting paper worksheets into interactive questionnaires, and adapting text and video 
components of PtDA DVDs into websites. While Internet-adapted PtDAs may have been rigorously tested and evaluated in 
their original format, It is important to consider whether the adapted version has been tested and evaluated as used on the 
Internet. 

May Include PtDAs that were specifically designed and tested for use on the Internet. Examples include websites designed 
to help patients with specific health care decisions by interactively tailoring information and support to their needs, or by 
providing opportunities for family members to participate in the discussion. Internet-based features may also include e-mail, 
discussion forums, blogs, or social media sites (e.g., Facebook, patient community websites) facilitated by clinicians, decision 
coaches, peers, or patient advocacy groups. 
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Table 2 Evaluative Characteristics of PtDAs Delivered on the Internet [20,33-43] 



Accessibility of the 
Technology 



Universality of the 
Technology 

Usability of the 
Technology 



Refers to the degree to which all people can access the Internet using whichever device they prefer (e.g., computer, 
laptop, tablet, or mobile phone) regardless of available dial-up/high-speed, Wi-Fi, or mobile phone Internet service. 
Dissemination strategies to maximize technology accessibility include providing both text-heavy and graphics-heavy 
versions of a PtDA, as well as versions for multiple screen sizes and mobile phones (known as responsive design). 

Refers to the degree to which the PtDA is accessible for men and women of all ages, races, ethnicities, religions, 
languages, and cultures. 

"...refers to how well users can learn and use a product to achieve their goals and how satisfied they are with that 
process" (ISO/AWI TR 9241-1 1, 1998) [43]. This definition includes a combination of factors, primarily focusing on five 
areas: 

1. Ease of Learning - How easy it is to do basic tasks the first time you see the website, 

2. Efficiency of Use - How quickly you can use the website once you know how, 

3. Memorability - How well you can remember how to use it the next time you visit, 

4. Error Frequency - How many errors are typically made in looking for information, and 

5. Satisfaction - How much you like using the website. 



needs assessments [19] prior to the development of the 
Internet-deUvered version; whether usability/accessibility/ 
field testing (from the perspectives of human computer 
interaction, user experience design, decision psychology, 
etc.) had been carried out prior to testing effectiveness; and 
the possible confounding effects of the selected delivery 
location and timing relative to the clinical appointment. 

A brief review of the literature since 2009 reveals several 
emerging efforts to develop and test Internet-delivered 
PtDAs. Additional file 2 Table S2 illustrates a selection of 
studies from decision support and health informatics pub- 
lications and presentations at scientific meetings from 
2010-2012. Similar to the earlier studies, results indicate 
that using an Internet-delivered PtDA: a) improved knowl- 
edge, preparation for decision making, and decisional con- 
flict scores; b) were acceptable to patients; and c) had 
variable effects on treatment preferences or the likelihood 
of receiving screening. These second-generation studies 



also began the exploration of interactivity and tailoring 
features, using health informatics and user-centered design 
methods during development and field-testing. Initial 
observations indicate that tailoring of clinical information 
and interactive deliberative guidance may improve deci- 
sion-making engagement and outcomes. 

However, these studies again point to several gaps in the 
empirical evidence. Additional studies are currently needed 
to test the role of a) specific Internet features (e.g., audio 
voice-over, interactive graphics, touch-screen data entry, 
etc.); and b) different dissemination and implementation 
strategies (e.g., delivery timing relative to clinic visits, publi- 
cally-available versus clinician-prescribed; use with/without 
a decision coach, integration with electronic health records, 
reach into rural communities, etc.). Subsequent studies 
may investigate the isolated effect of Internet-delivery of 
PtDAs when controlled for these factors, as well as the 
effect of match/mismatch between these factors and 



Table 3 Most Recent (2011) Cochrane Collaboration Review of PtDAs [30]: Randomized Controlled Trials that Involved 
Internet Delivery [72-75] 



Author, Year, 
Reference 
Number 



Clinical Context 



Study Design 



Primary Results 



Frosch, 2003, [72] PSA screening 



RCr, n = 226 randomized to view video during appointment 
at clinic or website at home before clinic visit 



Ruffin, 2007, [74] Colorectal Web, RCT, n = 1 74 men randomized to either informational website 
cancer screening or Colorectal Web 



Krist, 2007, [75] PSA screening Clustered RCT, n = 497 randomized to paper PtDA, website 

PtDA, or no pre-visit education 



Frosch, 2008, [73] PSA screening RCT, n = 61 1 randomized to web-based didactic PtDA, disease 

model + time trade-off exercise, both, or public PSA websites 



For both groups: 

- Knowledge scores were similar; 

- Ratings of effort required, convenience, 
and satisfaction were similar. 

Viewing the video at the clinic: 

- Increased the likelihood of viewing the 
complete PtDA 

Viewing Colorectal Web: 

- increased immediate reporting of preferred 
test, but no difference at 2, 8, or 24 weeks 

- increased screening 

Viewing either PtDA: 

- Increased decisional control 

- Increased knowledge scores 

- Decreased screening 

Knowledge scores were: 

- Highest for didactic PtDA 

- Lowest for public websites Post-PtDA 
screening preferences differed across groups. 
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patients' expectations. In order to address these gaps in evi- 
dence, multidisciplinary models are needed that include 
approaches from the fields of informatics, user-centered 
design, and human-computer interaction. Internet-deliv- 
ered PtDAs developed without appropriate technology 
testing (e.g., usability analyses, heuristic evaluations, card 
sorting, task analyses, etc.) may inadvertently introduce 
usability biases into the effectiveness trials. Hence, these 
studies highlight the importance of: a) involving key sta- 
keholders and users in the development and testing of 
Internet-delivered PtDAs; b) purposefully integrating 
development and evaluation methods firom the fields of 
health technology, decision support, and implementation 
science; and c) identifying short- and long-term research 
priorities. 

Lastiy, suites of online PtDAs are being rapidly developed 
by several academic health centers such as Dartmouth 
College, the Mayo Clinic, and the Universities of Ottawa, 
Cardiff, Hamburg, Sydney, etc., as well as by private PtDA 
developers (e.g.. Health Dialog, Healthwise, Informed 
Medical Decision Foundation), health insurers (e.g., Group 
Health, the U.S. Veterans Administration, Techniker 
Krankenkasse, etc.), and patient groups (e.g., Alzheimer's 
Association, CommonGround, PatientsLikeMe, etc.). As 
studies led by different disciplines (such as practitioners in 
decision support, health informatics, business leadership) 
and from perspectives (e.g., academic, governmental, 
private, and patient-led) converge, evidence may be system- 
atically reviewed and critically assessed to generate multi- 
disciplinary, evidence-based models, methods, and quality 
measures. 

Discussion 

Building on the updated definition, theoretical rationale, 
and empirical evidence, the following discussion highlights 
some emerging areas for research that can contribute 
towards developing measures and standards of quality for 
the Internet-delivery of PtDAs. 

Multidisciplinary perspectives on definition 

Given the broad definition of "delivering patient decision 
aids on the Internet", several subtypes of Internet delivery 
could be defined. These subtypes could be characterized 
according to their development, design, and usability char- 
acteristics using terms from health technology, decision 
science, and implementation science (see Tables 1, 2 and 
Additional fQe 1 Table SI) [22-24,31-43]. It is important to 
note that many of these terms are currently evolving areas 
of debate and research (e.g., the relationship between acces- 
sibility, universality, and usability), and/or overlap or vary 
by disciplines (e.g., the usability of the technology versus 
the usability of the tool within a decision support program). 
Therefore, these tables are not intended to present an 



exhaustive list, but may stimulate across-field discussion 
about terminology and assumptions. 

Theory development 

Social connectivity and patients' stories 

Online PtDAs are beginning to include features that 
allow users to share their stories and to connect virtually 
for support. These features add to the burgeoning range 
of distinctive capabilities associated with delivery on the 
Internet (such as virtual connectivity, interactivity, indivi- 
dualization, real-time evolution of information, socially- 
derived information, etc.) [31-33], and generate multiple 
sub-categories across a spectrum of types of online 
PtDAs. At one end of the spectrum, PtDAs designed 
using traditional decision support research methods may 
be adapted online to include patients' stories in more 
engaging formats (using, for example, interactive tables 
or icon arrays that illustrate risks and benefits, and are 
linked to a patient story for each icon) [44]. They may 
also add open discussion forums, links to social media 
sites, or online peer support and guidance in decision 
making [45]. At the other end of the spectrum, some 
websites that began as online support groups are now 
adding decision-making guides and tools for creating 
individualized risk estimates (e.g., entering one's personal 
health information and comparing it against socially- 
derived averages) [46]. 

Accordingly, research will need to consider the extent to 
which established theories (such as Social Development 
Theory, Elaboration Likelihood Model, etc.) inform the 
study of how people perceive, value, and use personal 
experiences that are shared virtually and that evolve in 
real-time [6,9]. On one hand, some communication and 
education theories support the use of written or videoed 
patient stories because their saliency augments the percep- 
tion of personalized information, and because qualitative 
studies indicate that patients frequently request such 
experiential information [33,47,48]. On the other hand, 
evidence is lacking regarding the potential effects and/or 
biases that might be introduced by such patient stories. 
These include, for example: social matching/mismatching 
between the individuals in the stories and the individuals 
viewing the decision aid; inadvertent story-induced misre- 
presentations, such as over/under-weighting of risks/bene- 
fits; and hidden story-induced framing biases [49-51]. 
These effects may be increased or decreased with the 
further addition of interactivity, virtual social connections, 
and perceptions of socially-derived "evidence". 

While the presentation of this kind of content carries a 
high potential for inadvertentiy presenting biased informa- 
tion to consumers, it also serves as an emerging social 
experiment highlighting the importance of shared infor- 
mation, connectivity, and the identification of outcomes 
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that may be of the greatest relevance to patients. Further 
research is needed to clarify: what types of socially-shared 
content are needed, preferred, and beneficial/harmful to 
patients; whether this information should be monitored/ 
managed in the context of PtDAs; and whether it can be 
collected and presented in neutral, unbiased, low literacy, 
and culturally-relevant formats. Results of such research 
could, in turn, inform quality metrics related to the incor- 
poration of social interactivity capabilities and socially- 
derived stories/evidence into Internet-delivered PtDAs. 
Behavior change 

As online PtDAs begin to be used in long-term health 
behavior change interventions (e.g., when choosing one's 
smoking cessation, weight loss, or substance abuse thera- 
pies), theories about the stages of behavior change will 
need to be tested in this new context of Internet-supported 
decision making. For example, the Trans-Theoretical 
Model related to Stages of Change has been used in the 
development of Internet-delivered resources for patients 
seeking to improve evidence-based healthy behaviors [8], 
but has not been used for preference-sensitive health care 
decision maldng. The role of Internet-delivered deliberative 
guidance may be more or less effective at specific stages in 
behavior change, and/or foster more continuous support 
across stages. 

In situations in which decision support is nested in 
motivational counseling (e.g., when a client first agrees to 
begin the behavioral change of smoking cessation, and 
then chooses among available smoking cessation thera- 
pies) [52], existing models will need to be tested under 
conditions in which in-person and online components are 
mixed or combined. For example, are the patient activa- 
tion and engagement effects observed during in-person 
decision support interventions maintained when Internet- 
delivered PtDAs are nested in virtual counsehng interven- 
tions (e.g., when using online motivational interviewing, 
problem solving therapy, or cognitive behavioral therapy)? 
How will long-term peer support groups be sustained 
online, and is additional training needed for peer leaders 
who facilitate these groups? Are additional privacy and 
security measures needed? Finally, what quality metrics 
are appropriate in this unique context? 
Equity 

Until Internet access and usage becomes universal, several 
issues in implementation theory and health equity need to 
be considered [14,18,20,53]. For example, "direct-to-con- 
sumer" delivery of PtDAs on the Internet is currently less 
feasible for communities that lack either high-speed access 
or cellular phone service [53]. However, insertion of a tele- 
health clinic in rural communities can extend health care 
access for patients who otherwise may be limited by travel 
times and costs, such as working adults who cannot attend 
daytime doctor visits, individuals confined to home by 
chronic medical conditions, and older adults who no 



longer drive [18]. Furthermore, current trends indicate 
that retired adults are the fastest growing group of Inter- 
net adopters and therefore may benefit from targeted deci- 
sion support programs on the Internet, but it remains 
unknown whether this trend is generational or situational 
(that is, whether the currently-employed generation will 
also increase their Internet use once they retire) [24]. 
Research is needed to support the short-, mid-, and long- 
term prioritization of development strategies for Internet- 
delivered PtDAs. 

In addition, emerging evidence suggests that there are 
population-wide shifts from personal computers to mobile 
smart phones, with disadvantaged groups becoming the 
largest users of smart phones for Internet access [27,28]. 
Insights gained from research in health informatics and in 
health communication can provide frameworks and tools 
for tailoring interventions to a user's characteristics and 
preferred Internet access media (e.g., computer, tablet, 
mobile phone, etc.) [36,54-56]. New efforts are being made 
to address health equity gaps by developing targeted/tai- 
lored mobile health websites and applications [53,57]. 
These frameworks will need to be applied and tested 
during the development of PtDAs. Additional research is 
specifically needed a) to develop methods and quality mea- 
sures for tailoring according to patients' preferred delib- 
erative styles (i.e., their preferred level information detail 
for making decisions, and their favored degree of interac- 
tive engagement with decision support), and b) to explore 
the effects of culturally-tailored experiential information 
about health care decision making (e.g., socially-derived 
patient stories). During this transition period in Internet 
use, theories about implementation and health equity can 
guide the Internet delivery of PtDAs, while taking into 
consideration a range of potential short-term and long- 
term strategies for shifting patient populations and evol- 
ving quality- assessment metrics. 
Chronic illness 

There is a need for interdisciplinary frameworks that can 
guide interventions to address the emerging needs and 
preferences of those patients with chronic illnesses who 
engage online with decision support and self-manage- 
ment strategies. Self-management theories support the 
use of technology for documenting personal experiences 
over time, and social learning theories support the sal- 
iency of shared experiences [33,47]. However, as patients 
are more frequently reporting their chronic illness 
experiences on the Internet [46], questions arise about 
whether and how to incorporate these rapidly-growing 
banks of patient-reported data into the theoretical frame- 
works underpinning decision support and evidence-based 
medicine. 

In chronic and progressive illness, decision support 
involves multiple decision points along the course of 
disease, multiple delivery settings (e.g., in hospitals. 
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community clinics, nursing homes, senior centers, etc.), 
and multiple intervention goals (e.g., cure, maintenance, 
palliation) that may include multiple decision makers (e.g., 
patient, local caregiver, long distance family members, 
clinical teams, legal proxies, etc.). Internet delivery offers 
some unique capabilities that could contribute positively 
to these complex processes [53,58,59]. For example, a ser- 
ies of related PtDAs can be delivered across the course of 
a progressive disease. For each decision point, a PtDA may 
offer support for urgent decision making or can facilitate 
advanced care planning. Decision support information and 
activities can be tailored to each user's role and preferred 
deliberative style at that point in time. For progressive dis- 
eases such as dementia, family members separated by long 
distances can engage in planning discussions using video 
chat features and shared family portals. Real-time docu- 
mentation on decision logs can track progression over 
time, and could be incorporated into electronic health 
records. 

However, unique challenges also arise — such as creating 
and maintaining multiple user portals, securing links to 
electronic health records, and maintaining data over time, 
etc. [60-62]. In addition, expectations for user-centered 
design may vary by generation and may need to evolve 
rapidly alongside advances in direct-to-consumer market- 
ing. Again, an interdisciplinary conceptual framework may 
be needed to create, maintain, and assess the quality of 
effective Internet- delivered PtDAs for chronic and pro- 
gressive conditions. 

Emerging areas in empirical evidence 

Interdisciplinary research efforts are also needed to 
address several gaps in the empirical evidence. These 
efforts could be considered as gaps in fundamental knowl- 
edge, in applied effort, and in research methodology. 
Fundamental investigations 

Fundamental studies conducted by decision scientists 
may focus on identifying the best methods for operatio- 
nalizing some or all components of an evidence-based 
decision support process using the Internet [32,63]. They 
may also seek to assess whether Internet-delivered PtDAs 
cause patients to engage differently at various points dur- 
ing their overall decision-making processes. Notably, 
fundamental research is needed to assess the "value 
added" of incorporating interactivity, personally-tailored 
information, interpersonal communication, patient stor- 
ies, and socially-generated experiential information. 
Compared to paper- or video-based PtDAs, do the unique 
features of Internet-based PtDAs (e.g., multimedia, inter- 
active activities, patient portals, social connectivity, etc.) 
improve patients' sense of preparation, self-efficacy, and 
decision quality for the immediate decision, as well as 
their ability to retain and transfer decision-making skills 
for future decisions? 



Health education scientists may seek to identify which 
technology features are most helpful for improving infor- 
mation comprehension, which graphic displays best con- 
vey risk/benefit information, and which interactive 
strategies can accurately assess whether a patient is well- 
informed [64,65]. Cognitive psychologists may investigate 
whether interactive PtDA websites can foster increased 
patient activation or increased communication with clini- 
cians, surrogate decision makers, and long-distance 
family members [59,66,67]. They may also assess which 
Internet PtDAs stand alone or need to be coupled with 
personal interaction, and whether the format and features 
of Internet-based PtDAs should vary by clinical context 
(e.g., preventive, acute, chronic, or end-of-life treatment 
decisions). 

Consumer health informatics researchers may also face 
continuing challenges in developing adaptive methods, 
tools, and measures to respond to the rapidly-changing 
field of health technologies. What are the emerging tech- 
nologies and features that are feasible, usable, and sus- 
tainable for clinicians, health care systems, and patient 
advocacy groups that wish to develop Internet-based 
decision support programs [60-62] Ongoing research will 
be necessary in order to create tools that are accessible 
and usable on multiple evolving devices (e.g., computers 
at the clinic, personal laptops, mobile phones, etc.) 
Health technology designers also face the challenge of 
allowing social connectivity and potential public sharing 
of experiential information for some patients, while 
ensuring the highest security for the protection of perso- 
nal health information for others [46,48,60,61]. 
Applied investigations 

From the perspective of decision-support scientists, a criti- 
cal first step in developing a PtDA is the "needs assess- 
ment", which characterizes the sources of difficulty that the 
relevant patient population has with the specific decision 
situation, and their needs and preferences for a decision 
support intervention (with or without a PtDA) [19]. When 
an Internet-delivered PtDA is considered, a needs assess- 
ment is particularly complex, since it should also assess 
diverse individuals' opinions about Internet-delivered clini- 
cal content, technology features, and implementation 
strategies [42,43,68]. Designers and developers of Internet- 
delivered PtDAs may seek to identify whether there are 
patterns in the preferred and effective features of such 
PtDAs, which then may be considered as standard attri- 
butes for acceptable quality. 

Alternatively, investigators may seek to determine 
whether specific components of PtDAs delivered on the 
Internet should be targeted or tailored to the clinical 
context (e.g., preventive, acute, chronic, and end-of-life 
treatment decisions), the users' socio-demographic char- 
acteristics (e.g., age, sex, race/ethnicity, culture, role in 
decision making), and/or the users' geographical locations 
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(e.g., urban/rural communities, international/national/ 
regional sites). For example, is "value added" when patients 
are guided to either culturally-neutral or culturally- 
matched socially-derived experiential information and/or 
support? Furthermore, Internet-delivered PtDAs allow 
patients greater opportunity to customize or "self-tailor" 
the level of clinical information detail, engagement in 
deliberative steps, and viewing of socially-derived experi- 
ential information according to their personal deliberative 
styles. However, this customization may conflict with 
existing quality measures for standardized information 
provision and decision support across the sequence of 
steps in the decision-making process. Research is needed 
to identify approaches that provide high-quality decision 
support that meets general standards while simultaneously 
allowing for varying degrees of targeting, tailoring, custo- 
mization, and personalization. 

Population-level investigations may explore the poten- 
tial added benefits and the possible risks of unintended 
harms generated by delivering PtDAs on the Internet, as 
well as the direct and indirect costs of developing, updat- 
ing, and maintaining Internet-delivered PtDAs, compared 
to using other media [20,61]. Health services researchers 
may assess whether or not the delivery of PtDAs on the 
Internet fosters increased health equity for disadvantaged 
communities, chronic care, surrogate decision makers, 
isolated adults (e.g., individuals with mobility disabilities, 
home-bound older adults), and/or dispersed families 
[20,59,65,66]. Investigators in implementation science 
and translational research may seek to determine the 
best practices for using Internet-based PtDAs to deliver 
"the right support to the right patient at the right time" 
[69-71]. 

Research methods 

While many research methods overlap across the relevant 
disciphnary perspectives, some diverge. Decision support 
researchers and health information technology researchers 
may include different steps in their design and evaluation 
process. Quality evaluators may use similar metrics, such 
as usability and acceptability, but may assess these in very 
different ways. Finally, they may focus on very different 
"high quality" outcomes. 

For example, decision support research methods that 
arise from medical research models include: systematic 
reviews of the literature; needs assessments; prototype 
development; field-tests; effectiveness trials; and dissemina- 
tion and implementation studies. During the field-testing 
of prototypes, usability and acceptability assessment may 
involve obtaining, from small groups of 10 - 30 patients, 
focus-group-based or questionnaire-based evaluations of 
the length, depth, level of detail, presence/freedom from 
bias, and interest generated by the clinical information and 
decision support steps [19,21]. Usability assessments may 
also include evaluations of the feasibility of using the new 



decision support tool in the flow of regular clinical care 
(e.g., before, during, or after a clinical consultation), or the 
feasibility of implementation across sites [20,63]. Other 
decision support outcomes may include measures of deci- 
sional conflict, preparation for decision making, decision 
quality, treatment choice, and adherence to chosen therapy 
[1,30,63]. 

Research methods that are specifically focused on 
Internet-delivered PtDAs could build on these medical 
research models by adding, early in the needs assess- 
ment and development phases, structured steps that 
specifically plan, analyze, design, and test the website 
itself [68]. These steps could include: assessing user 
characteristics and needs; developing narratives about 
target personas and key scenarios about site use; design- 
ing the content of the website; and incorporating itera- 
tive prototype-testing cycles [37-42]. For example, card 
sorting, wireframes, and task analyses may be used to 
structure the architecture of the content and to develop 
a prototype. Iterative cycles of prototype-testing with 
3-5 users may include interactive achievement testing and 
heuristic evaluations that track eye movements when 
skimming/reading the content, and that note, for example, 
the number of clicks, the frequency of errors, and page- 
loading speeds. Usability metrics may include ease of 
learning (e.g., how quickly a new user can complete basic 
tasks using the site), efficiency, error frequency, memor- 
ability, and satisfaction [22,43]. Additional outcome mea- 
sures may include continued website usage, accessibility 
for all types of users, and data privacy and security. 

Therefore, in order to consider the quality (that is, the 
conceptual appropriateness, the validity, the sensitivity, 
etc.) of various research methods that could be used to 
develop, test, and implement Internet-delivered PtDAs, 
integrated conceptual frameworks and models are needed. 
These frameworks could inform the development of a 
shared language for identifying rigorous research practices 
and "high quality" outcome measures. There is a distinc- 
tive scientific challenge here: how best to integrate these 
various approaches in order to systematically develop a 
strong evidence base about the design, testing, and imple- 
mentation of Internet-delivered PtDAs. 

Conclusion 

At present, there is notable theoretical justification for 
delivering some or all of the components of a PtDA using 
the Internet. Several theories in psychology, education, 
communication, and implementation science support the 
value of the interactive, multimedia, and accessibility fea- 
tures of Internet-delivered PtDAs. However, there is cur- 
rently a paucity of scientifically rigorous empirical studies 
investigating the role of Internet delivery for PtDAs, the 
usability of the user interfaces, and their effects in differing 
clinical, cultural, and decision-making contexts. 
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Accordingly, there is a rapidly expanding field of inter- 
disciplinary research opportunities. In particular, studies 
are needed to assess the role of tailored information and 
support, the optimal uses of the unique features of 
the Internet, and the potential benefits and harms of 
socially-generated information about patient stories. 
Equity studies may also assess implementation strategies 
that include both paper-based PtDAs in medical centers 
and Internet-delivered PtDAs in rural communities. 
Ultimately, cost-effectiveness and cost-benefit analyses 
are needed to compare Internet-, video-, and paper- 
based strategies for dissemination, maintenance, and 
sustainability. 

Finally, an interdisciplinary and internationally-repre- 
sentative approach is needed to identify best practices 
for a) needs assessments, b) user-centered development, 
c) usability assessments, d) field testing, d) clinical effec- 
tiveness, and e) broad and sustainable implementation 
strategies. As these gaps in the empirical evidence are 
addressed, the international scientific community may 
wish to revisit continually the criteria used to gauge the 
quality of Internet-delivered PtDAs. 
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